Image processing method and apparatus, and electronic device

ABSTRACT

An image processing method is provided. The method is configured to process the color-block image output by the image sensor. The high-frequency region of the color-block image is determined. A part of the color-block image within the high-frequency region is converted into a first image using a first interpolation algorithm. A part of the color-block image beyond the high-frequency region is converted into a second image using a second interpolation algorithm. The complexity of the second interpolation algorithm is less than that of the first interpolation algorithm. The first image and the second image are merged into a simulation image corresponding to the color-block image. An image processing apparatus and an electronic device are provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/800,233, filed on Nov. 1, 2017, which claims priority to ChinesePatent Application No. 201611079544.7, filed on Nov. 29, 2016. Theentire disclosures of the aforementioned applications are incorporatedherein by reference.

FIELD

The present disclosure relates to the imaging technology field, and moreparticularly to an image processing method, an image processingapparatus and an electronic device.

BACKGROUND

When an image is processed using a conventional image processing method,either the obtained image has a low resolution, or it takes a long timeand too much resource to obtain an image with high resolution, both ofwhich are inconvenient for users.

DISCLOSURE

The present disclosure aims to solve at least one of existing problemsin the related art to at least extent. Accordingly, the presentdisclosure provides an image processing method, an image processingapparatus and an electronic device.

Embodiments of the present disclosure provide an image processingmethod. The image processing method is configured to process acolor-block image output by an image sensor. The image sensor includesan array of photosensitive pixel units and an array of filter unitsarranged on the array of photosensitive pixel units. Each filter unitcorresponds to one photosensitive pixel unit, and each photosensitivepixel unit includes a plurality of photosensitive pixels. Thecolor-block image includes image pixel units arranged in a preset array.Each image pixel unit includes a plurality of original pixels. Eachphotosensitive pixel unit corresponds to one image pixel unit, and eachphotosensitive pixel corresponds to one original pixel. The imageprocessing method includes: determining a high-frequency region of thecolor-block image; converting a part of the color-block image within thehigh-frequency region into a first image using a first interpolationalgorithm, in which, the first image includes first simulation pixelsarranged in an array, and each photosensitive pixel corresponds to onefirst simulation pixel; converting a part of the color-block imagebeyond the high-frequency region into a second image using a secondinterpolation algorithm, in which, the second image includes secondsimulation pixels arranged in an array, and each photosensitive pixelcorresponds to one second simulation pixel, and a complexity of thesecond interpolation algorithm is less than that of the firstinterpolation algorithm; and merging the first image and the secondimage into a simulation image corresponding to the color-block image.

Embodiments of the present disclosure further provide an imageprocessing apparatus. The image processing apparatus is configured toprocess a color-block image output by an image sensor. The image sensorincludes an array of photosensitive pixel units and an array of filterunits arranged on the array of photosensitive pixel units. Each filterunit corresponds to one photosensitive pixel unit, and eachphotosensitive pixel unit includes a plurality of photosensitive pixels.The color-block image includes image pixel units arranged in a presetarray. Each image pixel unit includes a plurality of original pixels.Each photosensitive pixel unit corresponds to one image pixel unit, andeach photosensitive pixel corresponds to one original pixel. The imageprocessing apparatus includes a non-transitory computer-readable mediumcomprising computer-readable instructions stored thereon, and aninstruction execution system which is configured by the instructions toimplement at least one of a determining module, a first convertingmodule, a second converting module, and a merging module. Thedetermining module is configured to determine a high-frequency region ofthe color-block image. The first converting module is configured toconvert a part of the color-block image within the high-frequency regioninto a first image using a first interpolation algorithm. The firstimage includes first simulation pixels arranged in an array, and eachphotosensitive pixel corresponds to one first simulation pixel. Thesecond converting module is configured to convert a part of thecolor-block image beyond the high-frequency region into a second imageusing a second interpolation algorithm. The second image includes secondsimulation pixels arranged in an array, and each photosensitive pixelcorresponds to one second simulation pixel. A complexity of the secondinterpolation algorithm is less than that of the first interpolationalgorithm. The merging module is configured to merge the first image andthe second image into a simulation image corresponding to thecolor-block image.

Embodiments of the present disclosure provide an electronic device. Theelectronic device includes a housing, a processor, a memory, a circuitboard, a power supply circuit and an image sensor. The circuit board isenclosed by the housing. The processor and the memory are positioned onthe circuit board. The power supply circuit is configured to providepower for respective circuits or components of the electronic device.The image sensor is configured to output a color-block image. The imagesensor includes an array of photosensitive pixel units and an array offilter units arranged on the array of photosensitive pixel units. Eachfilter unit corresponds to one photosensitive pixel unit, and eachphotosensitive pixel unit includes a plurality of photosensitive pixels.The color-block image includes image pixel units arranged in a presetarray. Each image pixel unit includes a plurality of original pixels.Each photosensitive pixel unit corresponds to one image pixel unit, andeach photosensitive pixel corresponds to one original pixel. The memoryis configured to store executable program codes. The processor isconfigured to run a program corresponding to the executable programcodes by reading the executable program codes stored in the memory, toperform the image processing method according to the above embodimentsof the present disclosure.

Additional aspects and advantages of embodiments of the presentdisclosure will be given in part in the following descriptions, becomeapparent in part from the following descriptions, or be learned from thepractice of the embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects and advantages of embodiments of the presentdisclosure will become apparent and more readily appreciated from thefollowing descriptions made with reference to the drawings.

FIG. 1 is a flow chart of an image processing method according to anembodiment of the present disclosure.

FIG. 2 is a block diagram of an image sensor according to an embodimentof the present disclosure.

FIG. 3 is a schematic diagram of an image sensor according to anembodiment of the present disclosure.

FIG. 4 is a flow chart showing a process of determining a high-frequencyregion of a color-block image according to an embodiment of the presentdisclosure.

FIG. 5 is a schematic diagram showing frequency analysis regionsaccording to an embodiment of the present disclosure.

FIG. 6 is a flow chart showing a process of converting a part of acolor-block image into a first image according to an embodiment of thepresent disclosure.

FIG. 7 is a schematic diagram illustrating a circuit of an image sensoraccording to an embodiment of the present disclosure.

FIG. 8 is a schematic diagram of an array of filter units according toan embodiment of the present disclosure.

FIG. 9 is a schematic diagram of a merged image according to anembodiment of the present disclosure.

FIG. 10 is a schematic diagram of a color-block image according to anembodiment of the present disclosure.

FIG. 11 is a schematic diagram illustrating a process of converting acolor-block image into a first image according to an embodiment of thepresent disclosure.

FIG. 12 is a flow chart showing a process of converting a part of acolor-block image into a first image according to another embodiment ofthe present disclosure.

FIG. 13 is a flow chart showing a process of converting a part of acolor-block image into a first image according to another embodiment ofthe present disclosure.

FIG. 14 is a schematic diagram showing an image pixel unit of acolor-block image according to an embodiment of the present disclosure.

FIG. 15 is a flow chart showing a process of converting a part of acolor-block image into a first image according to another embodiment ofthe present disclosure.

FIG. 16 is a flow chart illustrating a process of converting a part of acolor-block image into a second image according to an embodiment of thepresent disclosure.

FIG. 17 is a schematic diagram illustrating a process of converting acolor-block image into a second image according to an embodiment of thepresent disclosure.

FIG. 18 is a block diagram of an image processing apparatus according toan embodiment of the present disclosure.

FIG. 19 is a block diagram of a determining module according to anembodiment of the present disclosure.

FIG. 20 is a block diagram of a first converting module according to anembodiment of the present disclosure.

FIG. 21 is a block diagram of a third determining unit in the firstconverting module according to an embodiment of the present disclosure.

FIG. 22 is a block diagram of a first converting module according toanother embodiment of the present disclosure.

FIG. 23 is a block diagram of a first converting module according toanother embodiment of the present disclosure.

FIG. 24 is a block diagram of a second converting module according to anembodiment of the present disclosure.

FIG. 25 is a block diagram of an electronic device according to anembodiment of the present disclosure.

EMBODIMENTS OF THE PRESENT DISCLOSURE

Reference will now be made in detail to exemplary embodiments, examplesof which are illustrated in the accompanying drawings, in which the sameor similar reference numbers throughout the drawings represent the sameor similar elements or elements having same or similar functions.Embodiments described below with reference to drawings are merelyexemplary and used for explaining the present disclosure, and should notbe understood as limitation to the present disclosure.

It is to be understood that phraseology and terminology used herein withreference to device or element orientation (such as, terms like“center”, “longitudinal”, “lateral”, “length”, “width”, “height”, “up”,“down”, “front”, “rear”, “left”, “right”, “vertical”, “horizontal”,“top”, “bottom”, “inside”, “outside”, “clockwise”, “anticlockwise”,“axial”, “radial”, “circumferential”) are only used to simplifydescription of the present invention, and do not indicate or imply thatthe device or element referred to must have or operated in a particularorientation. They cannot be seen as limits to the present disclosure.

Moreover, terms of “first” and “second” are only used for descriptionand cannot be seen as indicating or implying relative importance orindicating or implying the number of the indicated technical features.Thus, the features defined with “first” and “second” may comprise orimply at least one of these features. In the description of the presentdisclosure, “a plurality of” means two or more than two, unlessspecified otherwise.

In the present disclosure, unless specified or limited otherwise, theterms “mounted,” “connected,” “coupled,” “fixed” and the like are usedbroadly, and may be, for example, fixed connections, detachableconnections, or integral connections; may also be mechanical orelectrical connections; may also be direct connections or indirectconnections via intervening structures; may also be inner communicationsof two elements or interactions of two elements, which can be understoodby those skilled in the art according to specific situations.

In the present disclosure, unless specified or limited otherwise, astructure in which a first feature is “on” a second feature may includean embodiment in which the first feature directly contacts the secondfeature, and may also include an embodiment in which the first featureindirectly contacts the second feature via an intermediate medium.Moreover, a structure in which a first feature is “on”, “over” or“above” a second feature may indicate that the first feature is rightabove the second feature or obliquely above the second feature, or justindicate that a horizontal level of the first feature is higher than thesecond feature. A structure in which a first feature is “below”, or“under” a second feature may indicate that the first feature is rightunder the second feature or obliquely under the second feature, or justindicate that a horizontal level of the first feature is lower than thesecond feature.

Various embodiments and examples are provided in the followingdescription to implement different structures of the present disclosure.In order to simplify the present disclosure, certain elements andsettings will be described. However, these elements and settings areonly examples and are not intended to limit the present disclosure. Inaddition, reference numerals may be repeated in different examples inthe disclosure. This repeating is for the purpose of simplification andclarity and does not refer to relations between different embodimentsand/or settings. Furthermore, examples of different processes andmaterials are provided in the present disclosure. However, it would beappreciated by those skilled in the art that other processes and/ormaterials may be also applied.

In the related art, an image sensor includes an array of photosensitivepixel units and an array of filter units arranged on the array ofphotosensitive pixel unit. Each filter unit corresponds to and coversone photosensitive pixel unit, and each photosensitive pixel unitincludes a plurality of photosensitive pixels. When working, the imagesensor can be controlled to output a merged image, which can beconverted into a merged true-color image by an image processing methodand saved. The merged image includes an array of merged pixels, and aplurality of photosensitive pixels in a same photosensitive pixel unitare collectively outputted as one merged pixel. Thus, a signal-to-noiseratio of the merge image is increased. However, a resolution of themerged image is reduced.

Certainly, the image sensor also can be controlled to output a highpixel color-block image, which includes an array of original pixels, andeach photosensitive pixel corresponds to one original pixel. However,since a plurality of original pixels corresponding to a same filter unithave the same color, the resolution of the color-block image stillcannot be increased. Thus, the high pixel color-block image needs to beconverted into a high pixel simulation image by an interpolationalgorithm, in which the simulation image includes a Bayer array ofsimulation pixels. Then, the simulation image can be converted into asimulation true-color image by an image processing method and saved.However, the interpolation algorithm consumes resource and time, and thesimulation true-color image is not required in all scenes.

Thus, embodiments of the present disclosure provide a novel imageprocessing method.

Referring to FIG. 1, an image processing method is illustrated. Theimage processing method is configured to process a color-block imageoutput by an image sensor. As illustrated in FIGS. 2 and 3, the imagesensor 200 includes an array 210 of photosensitive pixel units and anarray 220 of filter units arranged on the array 210 of photosensitivepixel units. Each filter unit 220 a corresponds to one photosensitivepixel unit 210 a, and each photosensitive pixel unit 210 a includes aplurality of photosensitive pixels 212. The color-block image includesimage pixel units arranged in a preset array. Each image pixel unitincludes a plurality of original pixels. Each photosensitive pixel unit210 a corresponds to one image pixel unit, and each photosensitive pixel212 corresponds to one original pixel. The image processing methodincludes the followings.

At block 10, a high-frequency region of the color-block image isdetermined.

At block 20, a part of the color-block image within the high-frequencyregion is converted into a first image using a first interpolationalgorithm.

The first image includes first simulation pixels arranged in an array,and each photosensitive pixel 212 corresponds to one first simulationpixel.

At block 30, a part of the color-block image beyond the high-frequencyregion is converted into a second image using a second interpolationalgorithm.

The second image includes second simulation pixels arranged in an array,and each photosensitive pixel 212 corresponds to one second simulationpixel. A complexity of the second interpolation algorithm is less thanthat of the first interpolation algorithm.

At block 40, the first image and the second image are merged into asimulation image corresponding to the color-block image.

With the image processing method according to embodiments of the presentdisclosure, the first interpolation algorithm capable of improvingdistinguishability and resolution of the image is adopted for the partof the color-block image within the high-frequency region, and thesecond interpolation algorithm with complexity less than that of thefirst interpolation algorithm is adopted for the part of the color-blockimage beyond the high-frequency region, such that on one hand, SNR(signal to noise ratio), distinguishability and resolution of the imageare improved, thereby improving user experience, on other hand, timerequired for image processing is reduced.

In some implementations, the algorithm complexity includes the timecomplexity and the space complexity, and both the time complexity andthe space complexity of the second interpolation algorithm are less thanthose of the first interpolation algorithm. The time complexity isconfigured to measure a time consumed by the algorithm, and the spacecomplexity is configured to measure a storage space consumed by thealgorithm. If the time complexity is small, it indicates that thealgorithm consumes little time. If the space complexity is small, itindicates that the algorithm consumes little storage space. Thus, it isadvantageous to improve calculation speed by using the secondinterpolation algorithm, such that the shooting process is smooth, thusimproving the user experience.

Referring to FIG. 4, in some implementations, the act at block 10includes the followings.

At block 11, the color-block image is divided into a plurality offrequency analysis regions.

At block 12, a space frequency value of each of the plurality offrequency analysis regions is calculated.

At block 13, frequency analysis regions each with a space frequencyvalue conforming to a preset condition are merged into thehigh-frequency region.

In some embodiments, the preset condition includes: the space frequencyvalue being greater than a preset frequency threshold.

In detail, referring to FIG. 5, the color-block image can be dividedinto n*m frequency analysis regions. The space frequency value of eachfrequency analysis region is calculated and a preset frequency thresholdis set for each frequency analysis region, for example, frequencythresholds W_(1×1), W_(1×2), W_(1×3), . . . W_(n×m) can be set for then*m frequency analysis regions. Since the part located at edgesgenerally does not belong to the photographed object, the frequencythresholds of the regions close to the edges can be set to be smallvalues, while frequency thresholds of regions located in the center canbe set to be big values. The frequency thresholds can be set accordingto specific situations. When the space frequency value of a frequencyanalysis region is greater than the corresponding frequency threshold,the frequency analysis region can be determined as the high-frequencyregion.

Referring to FIG. 6, in some implementations, the act at block 20includes the followings.

At block 21, it is determined whether a color of a first simulationpixel is identical to that of an original pixel at a same position asthe first simulation pixel, if yes, an act at block 22 is executed,otherwise, an act at block 23 is executed.

At block 22, a pixel value of the original pixel is determined as apixel value of the first simulation pixel.

At block 23, the pixel value of the first simulation pixel is determinedaccording to a pixel value of an association pixel.

The association pixel is selected from an image pixel unit with a samecolor as the first simulation pixel and adjacent to an image pixel unitincluding the original pixel.

FIG. 7 is a schematic diagram illustrating a circuit of an image sensoraccording to an embodiment of the present disclosure. FIG. 8 is aschematic diagram of an array of filter units according to an embodimentof the present disclosure. FIGS. 2-3 and 7-8 are better viewed together.

Referring to FIGS. 2-3 and 7-8, the image sensor 200 according to anembodiment of the present disclosure includes an array 210 ofphotosensitive pixel units and an array 220 of filter units arranged onthe array 210 of photosensitive pixel units.

Further, the array 210 of photosensitive pixel units includes aplurality of photosensitive pixel units 210 a. Each photosensitive pixelunit 210 a includes a plurality of adjacent photosensitive pixels 212.Each photosensitive pixel 212 includes a photosensitive element 2121 anda transmission tube 2122. The photosensitive element 2121 may be aphotodiode, and the transmission tube 2122 may be a MOS transistor.

The array 220 of filter units includes a plurality of filter units 220a. Each filter unit 220 a corresponds to one photosensitive pixel unit210 a.

In detail, in some examples, the filter units are arranged in a Bayerarray. In other words, four adjacent filter units 220 a include one redfilter unit, one blue filter unit and two green filter units.

Each photosensitive pixel unit 210 a corresponds to a filter unit 220 awith a same color. If a photosensitive pixel unit 210 a includes nadjacent photosensitive elements 2121, one filter unit 220 a covers nphotosensitive elements 2121 in one photosensitive pixel unit 210 a. Thefilter unit 220 a may be formed integrally, or may be formed byassembling n separate sub filters.

In some implementations, each photosensitive pixel unit 210 a includesfour adjacent photosensitive pixels 212. Two adjacent photosensitivepixels 212 collectively form one photosensitive pixel subunit 2120. Thephotosensitive pixel subunit 2120 further includes a source follower2123 and an analog-to-digital converter 2124. The photosensitive pixelunit 210 a further includes an adder 213. A first electrode of eachtransmission tube 2122 in the photosensitive pixel subunit 2120 iscoupled to a cathode electrode of a corresponding photosensitive element2121. Second electrodes of all the transmission tubes 2122 in thephotosensitive pixel subunit 2120 are collectively coupled to a gateelectrode of the source follower 2123 and coupled to ananalog-to-digital converter 2124 via the source electrode of the sourcefollower 2123. The source follower 2123 may be a MOS transistor. Twophotosensitive pixel subunits 2120 are coupled to the adder 213 viarespective source followers 2123 and respective analog-to-digitalconverters 2124.

In other words, four adjacent photosensitive elements 2121 in onephotosensitive pixel unit 210 a of the image sensor 200 according to anembodiment of the present disclosure collectively use one filter unit220 a with a same color as the photosensitive pixel unit. Eachphotosensitive element 2121 is coupled to a transmission tube 2122correspondingly. Two adjacent photosensitive elements 2121 collectivelyuse one source follower 2123 and one analog-digital converter 2124. Fouradjacent photosensitive elements 2121 collectively use one adder 213.

Further, four adjacent photosensitive elements 2121 are arranged in a2-by-2 array. Two photosensitive elements 2121 in one photosensitivepixel subunit 2120 can be in a same row.

During an imaging process, when two photosensitive pixel subunits 2120or four photosensitive elements 2121 covered by a same filter unit 220 aare exposed simultaneously, pixels can be merged, and the merged imagecan be outputted.

In detail, the photosensitive element 2121 is configured to convertlight into charge, and the charge is proportional to an illuminationintensity. The transmission tube 2122 is configured to control a circuitto turn on or off according to a control signal. When the circuit isturned on, the source follower 2123 is configured to convert the chargegenerated through light illumination into a voltage signal. Theanalog-to-digital converter 2124 is configured to convert the voltagesignal into a digital signal. The adder 213 is configured to add twodigital signals to output.

Referring to FIG. 9, take an image sensor 200 of 16M as an example. Theimage sensor 200 according to an embodiment of the present disclosurecan merge photosensitive pixels 212 of 16M into photosensitive pixels of4M, i.e., the image sensor 200 outputs the merged image. After themerging, the photosensitive pixel 212 quadruples in size, such that thephotosensibility of the photosensitive pixel 212 is increased. Inaddition, since most part of noise in the image sensor 200 is random,there may be noise points at one or two pixels. After fourphotosensitive pixels 212 are merged into a big photosensitive pixel212, an effect of noise points on the big photosensitive pixel isreduced, i.e., the noise is weakened and SNR (signal-to-noise ratio) isimproved.

However, when the size of the photosensitive pixel 212 is increased, thepixel value is decreased, and thus the resolution of the merged image isdecreased.

During an imaging process, when four photosensitive elements 2121covered by a same filter unit 220 a are exposed in sequence, acolor-block image is output after an image processing.

In detail, the photosensitive element 2121 is configured to convertlight into charge, and the charge is proportional to an illuminationintensity. The transmission tube 2122 is configured to control a circuitto turn on or off according to a control signal. When the circuit isturned on, the source follower 2123 is configured to convert the chargegenerated through light illumination into a voltage signal. Theanalog-to-digital converter 2124 is configured to convert the voltagesignal into a digital signal for being processed.

Referring to FIG. 10, take an image sensor 200 of 16M as an example. Theimage sensor according to an embodiment of the present disclosure canoutput photosensitive pixels 212 of 16M, i.e., the image sensor 200outputs the color-block image. The color-block image includes imagepixel units. The image pixel unit includes original pixels arranged in a2-by-2 array. The size of the original pixel is the same as that of thephotosensitive pixel 212. However, since a filter unit 220 a coveringfour adjacent photosensitive elements 2121 has a same color (i.e.,although four photosensitive elements 2121 are exposed respectively, thefilter unit 220 a covering the four photosensitive elements has a samecolor), four adjacent original pixels in each image pixel unit of theoutput image have a same color, and thus the resolution of the imagecannot be increased.

The image processing method according to an embodiment of the presentdisclosure is able to process the output color-block image to obtain asimulation image.

In some embodiments, when a merged image is output, four adjacentphotosensitive pixels 212 with the same color can be output as onemerged pixel. Accordingly, four adjacent merged pixels in the mergedimage can be considered as being arranged in a typical Bayer array, andcan be processed directly to output a merged true-color image. When acolor-block image is output, each photosensitive pixel 212 is outputseparately. Since four adjacent photosensitive pixels 212 have a samecolor, four adjacent original pixels in an image pixel unit have a samecolor, which form an untypical Bayer array. However, the pixels with theuntypical Bayer array cannot be directly processed. In other words, whenthe image sensor 200 adopts a same apparatus for processing the image,in order to realize a compatibility of the true-color image outputsunder two modes (i.e., the merged true-color image under a merged modeand the simulation true-color image under a color-block mode), it isrequired to convert the color-block image into the simulation image, orto convert the image pixel unit in an untypical Bayer array into pixelsarranged in the typical Bayer array.

The simulation image includes simulation pixels arranged in the Bayerarray. Each photosensitive pixel corresponds to one simulation pixel.One simulation pixel in the simulation image corresponds to an originalpixel located at the same position as the simulation pixel and in thecolor-block image. According to embodiments of the present disclosure,the simulation image is merged by the first image and the second image.

By using the first interpolation algorithm, the part of the color-blockimage within the high-frequency region can be converted into the firstimage. The first image includes first simulation pixels arranged in anarray and each photosensitive pixel corresponds to one first simulationpixel.

FIG. 11 is a schematic diagram illustrating a process of converting acolor-block image into a first image according to an embodiment of thepresent disclosure.

Referring to FIG. 11, for the first simulation pixels R3′3′ and R5′5′,the corresponding original pixels are R33 and B55.

When the first simulation pixel R3′3′ is obtained, since the firstsimulation pixel R3′3′ has the same color as the corresponding originalpixel R33, the pixel value of the original pixel R33 is directlydetermined as the pixel value of the first simulation pixel R3′3′ duringconversion.

When the first simulation pixel R5′5′ is obtained, since the firstsimulation pixel R5′5′ has a color different from that of thecorresponding original pixel B55, the pixel value of the original pixelB55 cannot be directly determined as the pixel value of the firstsimulation pixel R5′5′, and it is required to calculate the pixel valueof the first simulation pixel R5′5′ according to an association pixel ofthe first simulation pixel R5′5′ by a first interpolation algorithm.

It should be noted that, a pixel value of a pixel mentioned in thecontext should be understood in a broad sense as a color attribute valueof the pixel, such as a color value.

There may be more than one association pixel unit for each firstsimulation pixel, for example, there may be four association pixelunits, in which the association pixel units have the same color as thefirst simulation pixel and are adjacent to the image pixel unitincluding the original pixel at the same position as the firstsimulation pixel.

It should be noted that, “adjacent” here should be understood in a broadsense. Take FIG. 11 as an example, the first simulation pixel R5′5′corresponds to the original pixel B55. The image pixel units 400, 500,600 and 700 are selected as the association pixel units, but other redimage pixel units far away from the image pixel unit where the originalpixel B55 is located are not selected as the association pixel units. Ineach association pixel unit, the red original pixel closest to theoriginal pixel B55 is selected as the association pixel, which meansthat the association pixels of the first simulation pixel R5′5′ includethe original pixels R44, R74, R47 and R77. The first simulation pixelR5′5′ is adjacent to and has the same color as the original pixels R44,R74, R47 and R77.

In different cases, the original pixels can be converted into the firstsimulation pixels in different ways, thus converting the color-blockimage into the first image. Since the filters in the Bayer array areadopted when shooting the image, the SNR of the image is improved.During the image processing procedure, the interpolation processing isperformed on the color-block image by the first interpolation algorithm,such that the distinguishability and resolution of the image can beimproved.

Referring to FIG. 12, in some implementations, the act at block 23(i.e., determining the pixel value of the first simulation pixelaccording to the pixel value of the association pixel) includes thefollowings.

At block 231, a change of the color of the first simulation pixel ineach direction of at least two directions is calculated according to thepixel value of the association pixel.

At block 232, a weight in each direction of the at least two directionsis calculated according to the change.

At block 233, the pixel value of the first simulation pixel iscalculated according to the weight and the pixel value of theassociation pixel.

In detail, the first interpolation algorithm is realized as follows:with reference to energy changes of the image in different directionsand according to weights of the association pixels in differentdirections, the pixel value of the first simulation pixel is calculatedby a linear interpolation. From the direction having a smaller energychange, it can get a higher reference value, i.e., the weight for thisdirection in the interpolation is high.

In some examples, for sake of convenience, only the horizontal directionand the vertical direction are considered.

The pixel value of the first simulation pixel R5′5′ is obtained by aninterpolation based on the original pixels R44, R74, R47 and R77. Sincethere is no original pixel with a same color as the simulation pixel(i.e., R) in the horizontal direction and the vertical direction of theoriginal pixel R55 corresponding the first simulation pixel R5′5′, acomponent of this color (i.e., R) in each of the horizontal directionand the vertical direction is calculated according to the associationpixels. The components in the horizontal direction are R45 and R75, andthe components in the vertical direction are R54 and R57. All thecomponents can be calculated according to the original pixels R44, R74,R47 and R77.

In detail, R45=R44*⅔+R47*⅓, R75=⅔*R74+⅓*R77, R54=⅔*R44+⅓*R74,R57=⅔*R47+⅓*R77.

The change of color and the weight in each of the horizontal directionand the vertical direction are calculated respectively. In other words,according to the change of color in each direction, the reference weightin each direction used in the interpolation is determined. The weight inthe direction with a small change is high, while the weight in thedirection with a big change is low. The change in the horizontaldirection is X1=|R45−R75|. The change in the vertical direction isX2=|R54−R57|, W1=X1/(X1+X2), W2=X2/(X1+X2).

After the above calculation, the pixel value of the first simulationpixel R5′5′ can be calculated asR5′5′=(⅔*R45+⅓*R75)*W2+(⅔*R54+⅓*R57)*W1. It can be understood that, ifX1>X2, then W1>W2. The weight in the horizontal direction is W2, and theweight in the vertical direction is W1, vice versa.

Accordingly, the pixel value of the first simulation pixel can becalculated by the first interpolation algorithm. After the calculationson the association pixels, the original pixels can be converted into thefirst simulation pixels arranged in the typical Bayer array. In otherwords, four adjacent first simulation pixels arranged in the 2-by-2array include one red first simulation pixel, two green first simulationpixels and one blue first simulation pixel.

It should be noted that, the first interpolation algorithm is notlimited to the above-mentioned method, in which only the pixel values ofpixels with a same color as the simulation pixel in the verticaldirection and the horizontal direction are considered during calculatingthe pixel value of the first simulation pixel. In other embodiments,pixel values of pixels with other colors can also be considered.

Referring to FIG. 13, in some embodiments, before the act at block 23,the method further includes performing a white-balance compensation onthe color-block image, as illustrated at block 24.

Accordingly, after the act at 23, the method further includes performinga reverse white-balance compensation on the first image, as illustratedat block 25.

In detail, in some examples, when converting the color-block image intothe first image, in the first interpolation algorithm, the red and bluefirst simulation pixels not only refer to the color weights of originalpixels having the same color as the first simulation pixels, but alsorefer to the color weights of original pixels with the green color.Thus, it is required to perform the white-balance compensation beforethe interpolation to exclude an effect of the white-balance in the firstinterpolation algorithm. In order to avoid the white-balance of thecolor-block image, it is required to perform the reverse white-balancecompensation after the first interpolation algorithm according to gainvalues of the red, green and blue colors in the compensation.

In this way, the effect of the white-balance in the first interpolationalgorithm can be excluded, and the simulation image obtained after theinterpolation can keep the white-balance of the color-block image.

Referring to FIG. 13 again, in some implementations, before the act atblock 23, the method further includes performing a bad-pointcompensation on the color-block image, as illustrated at block 26.

It can be understood that, limited by the manufacturing process, theremay be bad points in the image sensor 200. The bad point presents a samecolor all the time without varying with the photosensibility, whichaffects quality of the image. In order to ensure an accuracy of theinterpolation and prevent from the effect of the bad points, it isrequired to perform the bad-point compensation before the firstinterpolation algorithm is performed.

In detail, during the bad-point compensation, the original pixels aredetected. When an original pixel is detected as the bad point, thebad-point compensation is performed according to pixel values of otheroriginal pixels in the image pixel unit where the original pixel islocated.

In this way, the effect of the bad point on the interpolation can beavoided, thus improving the quality of the image.

Referring to FIG. 13 again, in some implementations, before the act atblock 23, the method includes performing a crosstalk compensation on thecolor-block image, as illustrated at block 27.

In detail, four photosensitive pixels 212 in one photosensitive pixelunit 210 a cover the filters with the same color, and the photosensitivepixels 212 have differences in photosensibility, such that fixedspectrum noise may occur in pure-color areas in the first true-colorimage outputted after converting the first image and the quality of theimage may be affected. Therefore, it is required to perform thecrosstalk compensation.

As explained above, in order to perform the crosstalk compensation, itis required to obtain the compensation parameters during themanufacturing process of the image sensor, and to store the parametersrelated to the crosstalk compensation into the storage of the imagesensor or the electronic device provided with the image sensor, such asthe mobile phone or tablet computer.

The preset luminous environment, for example, may include an LED uniformplate having a color temperature of about 5000K and a brightness ofabout 1000 lux. The imaging parameters may include a gain value, ashutter value and a location of a lens. After setting the relatedparameters, the crosstalk compensation parameters can be obtained.

During the process, multiple color-block images are obtained using thepreset imaging parameters in the preset luminous environment, andcombined into one combination color-block image, such that the effect ofnoise caused by using a single color-block image as reference can bereduced.

Referring to FIG. 14, take the image pixel unit Gr as an example. Theimage pixel unit Gr includes original pixels Gr1, Gr2, Gr3 and Gr4. Thepurpose of the crosstalk compensation is to adjust the photosensitivepixels which may have different photosensibilities to have the samephotosensibility. An average pixel value of the image pixel unit isGr_avg=(Gr1+Gr2+Gr3+Gr4)/4, which represents an average level ofphotosensibilities of the four photosensitive pixels. By configuring theaverage value as a reference value, ratios of Gr1/Gr_avg, Gr2/Gr_avg,Gr3/Gr_avg and Gr4/Gr_avg are calculated. It can be understood that, bycalculating a ratio of the pixel value of each original pixel to theaverage pixel value of the image pixel unit, a deviation between eachoriginal pixel and the reference value can be reflected. Four ratios canbe recorded in a storage of a related device as the compensationparameters, and can be retrieved during the imaging process tocompensate for each original pixel, thus reducing the crosstalk andimproving the quality of the image.

Generally, after setting the crosstalk compensation parameters,verification is performed on the parameters to determine the accuracy ofthe parameters.

During the verification, a color-block image is obtained with the sameluminous environment and same imaging parameters as the preset luminousenvironment and the preset imaging parameters, and the crosstalkcompensation is performed on the color-block image according to thecalculated compensation parameters to calculate compensated Gr′_avg,Gr′1/Gr′_avg, Gr′2/Gr′_avg, Gr′3/Gr′_avg and Gr′4/Gr′_avg. The accuracyof parameters can be determined according to the calculation resultsfrom a macro perspective and a micro perspective. From the microperspective, when a certain original pixel after the compensation stillhas a big deviation which is easy to be sensed by the user after theimaging process, it means that the parameters are not accurate. From themacro perspective, when there are too many original pixels withdeviations after the compensation, the deviations as a whole can besensed by the user even if a single original pixel has a smalldeviation, and in this case, the parameters are also not accurate. Thus,a ratio threshold can be set for the micro perspective, and anotherratio threshold and a number threshold can be set for the macroperspective. In this way, the verification can be performed on thecrosstalk compensation parameters to ensure the accuracy of thecompensation parameters and to reduce the effect of the crosstalk on thequality of the image.

Referring to FIG. 15, in some implementations, after the act at block23, the method further includes performing at least one of a mirrorshape correction, a demosaicking processing, a denoising processing andan edge sharpening processing on the first image, as illustrated atblock 28.

It can be understood that, after the color-block image is converted intothe first image, the first simulation pixels are arranged in the typicalBayer array. The first image can be processed, during which, the mirrorshape correction, the demosaicking processing, the denoising processingand the edge sharpening processing are included.

For the part of the color-block image beyond the high-frequency region,it is required to process this part using the second interpolationalgorithm. By using the second interpolation algorithm, the part of thecolor-block image beyond the high-frequency region can be converted intothe second image. The second image includes second simulation pixelsarranged in an array, and each photosensitive pixel corresponds to onesecond simulation pixel.

FIG. 16 is a flow chart illustrating a process of converting a part of acolor-block image into a second image according to an embodiment of thepresent disclosure.

Referring to FIG. 16, in some implementations, the act at block 30includes the followings.

At block 31, an average pixel value of each image pixel unit of thecolor-block image is calculated.

At block 32, it is determined whether a color of a second simulationpixel is identical to that of an original pixel at a same position asthe second simulation pixel, if yes, an act at block 33 is executed,otherwise, an act at block 34 is executed.

At block 33, an average pixel value of an image pixel unit including theoriginal pixel is determined as a pixel value of the second simulationpixel.

At block 34, an average pixel value of an image pixel unit with a samecolor as the second simulation pixel and adjacent to an image pixel unitincluding the original pixel is determined as a pixel value of thesecond simulation pixel.

In detail, referring to FIG. 17, an average pixel value for each imagepixel unit is calculated as follows: Ravg=(R1+R2+R3+R4)/4,Gravg=(Gr1+Gr2+Gr3+Gr4)/4, Gbavg=(Gb1+Gb2+Gb3+Gb4)/4, andBavg=(B1+B2+B3+B4)/4. In this case, the pixel value of each of R11, R12,R21 and R22 is Ravg, the pixel value of each of Gr31, Gr32, Gr41 andGr42 is Gravg, the pixel value of each of Gb13, Gb14, Gb23 and Gb24 isGbavg, and the pixel value of each of B33, B34, B43 and B44 is Bavg.Taking the second simulation pixel B22 as an example, the original pixelcorresponding to the second simulation pixel B22 is R22, which has adifferent color from that of B22, such that the average pixel value Bavgof the image pixel unit (including the original pixels B33, B34, B43 andB44) with the same color (blue) as B22 and adjacent to the image pixelunit including B22 is determined as the pixel value of B22. Similarly,the pixel value of each second simulation pixel with other colors can bedetermined using the second interpolation algorithm.

By using the second interpolation algorithm, the complexity of theprocess in which an untypical Bayer array is converted into a typicalBayer array is small. The distinguishability of the second image canalso be improved by the second interpolation algorithm, but the effectof the second image is poorer than that of the first image generatedusing the first interpolation algorithm. Therefore, the firstinterpolation algorithm is used to process the part of the color-blockimage within the high-frequency region and the second interpolationalgorithm is used to process the part of the color-block image beyondthe high-frequency region, thus improving the distinguishability andeffect of the main part of the image, improving the user experience andreducing the time for processing the image.

In another aspect, the present disclosure also provides an imageprocessing apparatus.

FIG. 18 is a block diagram of an image processing apparatus according toan embodiment of the present disclosure. Referring to FIG. 18 and FIGS.2-3 and 7-8, an image processing apparatus 4000 is illustrated. Theimage processing apparatus 4000 is configured to process a color-blockimage output by an image sensor 200. As illustrated above, the imagesensor 200 includes an array 210 of photosensitive pixel units and anarray 220 of filter units arranged on the array 210 of photosensitivepixel units. Each filter unit 220 a corresponds to one photosensitivepixel unit 210 a, and each photosensitive pixel unit 210 a includes aplurality of photosensitive pixels 212. The color-block image includesimage pixel units arranged in a preset array. Each image pixel unitincludes a plurality of original pixels. Each photosensitive pixel unit210 a corresponds to one image pixel unit, and each photosensitive pixel212 corresponds to one original pixel. The image processing apparatus4000 includes a non-transitory computer-readable medium 4600 and aninstruction execution system 4800. The non-transitory computer-readablemedium 4600 includes computer-executable instructions stored thereon. Asillustrated in FIG. 18, the non-transitory computer-readable medium 4600includes a plurality of program modules, including a determining module410, a first converting module 420, a second converting module 430 and amerging module 440. The instruction execution system 4800 is configuredby the instructions stored in the medium 4600 to implement the programmodules.

The determining module 410 is configured to determine a high-frequencyregion of the color-block image. The first converting module 420 isconfigured to convert a part of the color-block image within thehigh-frequency region into a first image using a first interpolationalgorithm. The first image includes first simulation pixels arranged inan array, and each photosensitive pixel 212 corresponds to one firstsimulation pixel. The second converting module 430 is configured toconvert a part of the color-block image beyond the high-frequency regioninto a second image using a second interpolation algorithm. The secondimage includes second simulation pixels arranged in an array, and eachphotosensitive pixel 212 corresponds to one second simulation pixel. Acomplexity of the second interpolation algorithm is less than that ofthe first interpolation algorithm. The merging module 440 is configuredto merge the first image and the second image into a simulation imagecorresponding to the color-block image.

In other words, the act at block 10 can be implemented by thedetermining module 410. The act at block 20 can be implemented by thefirst converting module 420. The act at block 30 can be implemented bythe second converting module 430. The act at block 40 can be implementedby the merging module 440.

With the image processing apparatus according to embodiments of thepresent disclosure, the first interpolation algorithm capable ofimproving distinguishability and resolution of the image is adopted forthe part of the color-block image within the high-frequency region, andthe second interpolation algorithm with complexity less than that of thefirst interpolation algorithm is adopted for the part of the color-blockimage beyond the high-frequency region, such that on one hand, SNR(signal to noise ratio), distinguishability and resolution of the imageare improved, thereby improving user experience, on other hand, timerequired for image processing is reduced.

Referring to FIG. 19, the determining module 410 includes a dividingunit 411, a first calculating unit 412, and a merging unit 413. Thedividing unit 411 is configured to divide the color-block image into aplurality of frequency analysis regions. The first calculating unit 412is configured to calculate a space frequency value of each of theplurality of frequency analysis regions. The merging unit 413 isconfigured to merge frequency analysis regions each with a spacefrequency value conforming to a preset condition into the high-frequencyregion.

In other words, the act at block 11 can be implemented by the dividingunit 411. The act at block 12 can be implemented by the firstcalculating unit 412. The act at block 13 can be implemented by themerging unit 413.

In some implementations, the preset condition includes: the spacefrequency value being greater than a preset frequency threshold.

Referring to FIG. 20, in some implementations, the first convertingmodule 420 includes a first determining unit 421, a second determiningunit 422 and a third determining unit 423. The act at block 21 can beimplemented by the first determining unit 421. The act at block 22 canbe implemented by the second determining unit 422. The act at block 23can be implemented by the third determining unit 423. In other words,the first determining unit 421 is configured to determine whether acolor of a first simulation pixel is identical to that of an originalpixel at a same position as the first simulation pixel. The seconddetermining unit 422 is configured to determine a pixel value of theoriginal pixel as a pixel value of the first simulation pixel when thecolor of the first simulation pixel is identical to that of the originalpixel at the same position as the first simulation pixel. The thirddetermining unit 423 is configured to determine the pixel value of thefirst simulation pixel according to a pixel value of an associationpixel when the color of the first simulation pixel is different fromthat of the original pixel at the same position as the first simulationpixel. The association pixel is selected from an image pixel unit with asame color as the first simulation pixel and adjacent to an image pixelunit comprising the original pixel.

Referring to FIG. 21, in some implementations, the third determiningunit 423 further includes a first calculating subunit 4231, a secondcalculating subunit 4232 and a third calculating subunit 4233. The actat block 231 can be implemented by the first calculating subunit 4231.The act at block 232 can be implemented by the second calculatingsubunit 4232. The act at block 233 can be implemented by the thirdcalculating subunit 4233. In other words, the first calculating subunit4231 is configured to calculate a change of the color of the firstsimulation pixel in each direction of at least two directions accordingto the pixel value of the association pixel. The second calculatingsubunit 4232 is configured to calculate a weight in each direction ofthe at least two directions according to the change. The thirdcalculating subunit 4233 is configured to calculate the pixel value ofthe first simulation pixel according to the weight and the pixel valueof the association pixel.

Referring to FIG. 22, in some implementations, the first convertingmodule 420 further includes a first compensating unit 424 and arestoring unit 425. The act at block 24 can be implemented by the firstcompensating unit 424. The act at block 25 can be implemented by therestoring unit 425. In other words, the first compensating unit 424 isconfigured to perform a white-balance compensation on the color-blockimage. The restoring unit 425 is configured to perform a reversewhite-balance compensation on the first image.

In some implementations, the first converting module 420 furtherincludes a second compensating unit 426. The act at block 26 can beimplemented by the second compensating unit 426. In other words, thesecond compensating unit 426 is configured to perform a bad-pointcompensation on the color-block image.

In some implementations, the first converting module 420 furtherincludes a third compensating unit 427. The act at block 27 can beimplemented by the third compensating unit 427. In other words, thethird compensating unit 427 is configured to perform a crosstalkcompensation on the color-block image.

FIG. 23 is a block diagram of a first converting module according toanother embodiment of the present disclosure. Referring to FIG. 23, insome implementations, the first converting module 420 includes aprocessing unit 428. The act at block 28 can be implemented by theprocessing unit 428. In other words, the processing unit 428 isconfigured to perform at least one of a mirror shape correction, ademosaicking processing, a denoising processing and an edge sharpeningprocessing on the first image.

Referring to FIG. 24, in some implementations, the second convertingmodule 430 includes a second calculating unit 431, a fourth determiningunit 432, a fifth determining unit 433 and a sixth determining unit 434.The second calculating unit 431 is configured to calculate an averagepixel value of each image pixel unit of the color-block image. Thefourth determining unit 432 is configured to determine whether a colorof a second simulation pixel is identical to that of an original pixelat a same position as the second simulation pixel. The fifth determiningunit 433 is configured to determine an average pixel value of an imagepixel unit including the original pixel as a pixel value of the secondsimulation pixel, when the color of the second simulation pixel isidentical to that of the original pixel at the same position as thesecond simulation pixel. The sixth determining unit 434 is configured todetermine an average pixel value of an image pixel unit with a samecolor as the second simulation pixel and adjacent to an image pixel unitincluding the original pixel as a pixel value of the second simulationpixel, when the color of the second simulation pixel is different fromthat of the original pixel at the same position as the second simulationpixel. In other words, the act at block 31 is implemented by the secondcalculating unit 431. The act at block 32 is implemented by the fourthdetermining unit 432. The act at block 33 is implemented by the fifthdetermining unit 433. The act at block 34 is implemented by the sixthdetermining unit 434.

The present disclosure also provides an electronic device.

FIG. 25 is a block diagram of an electronic device 1000 according to anembodiment of the present disclosure. Referring to FIG. 25, theelectronic device 1000 of the present disclosure includes a housing1001, a processor 1002, a memory 1003, a circuit board 1006, a powersupply circuit 1007 and an image sensor 200, The circuit board 1006 isenclosed by the housing 1001. The processor 1002 and the memory 1003 arepositioned on the circuit board 1006. The power supply circuit 1007 isconfigured to provide power for respective circuits or components of theelectronic device 1000. The memory 1003 is configured to storeexecutable program codes. As illustrated above, the image sensor 200includes an array 210 of photosensitive pixel units and an array 220 offilter units arranged on the array 210 of photosensitive pixel units.Each filter unit 220 a corresponds to one photosensitive pixel unit 210a, and each photosensitive pixel unit 210 a includes a plurality ofphotosensitive pixels 212. The color-block image includes image pixelunits arranged in a preset array. Each image pixel unit includes aplurality of original pixels. Each photosensitive pixel unit 210 acorresponds to one image pixel unit, and each photosensitive pixel 212corresponds to one original pixel.

The processor 1002 is configured to run a program corresponding to theexecutable program codes by reading the executable program codes storedin the memory 1003, to perform following operations: determining ahigh-frequency region of the color-block image; converting a part of thecolor-block image within the high-frequency region into a first imageusing a first interpolation algorithm, in which, the first imageincludes first simulation pixels arranged in an array, and eachphotosensitive pixel corresponds to one first simulation pixel;converting a part of the color-block image beyond the high-frequencyregion into a second image using a second interpolation algorithm, inwhich, the second image includes second simulation pixels arranged in anarray, and each photosensitive pixel corresponds to one secondsimulation pixel, and a complexity of the second interpolation algorithmis less than that of the first interpolation algorithm; and merging thefirst image and the second image into a simulation image correspondingto the color-block image.

In some implementations, the image sensor includes a front camera or areal camera (not illustrated in FIG. 25).

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to determine ahigh-frequency region of the color-block image by acts of: dividing thecolor-block image into a plurality of frequency analysis regions;calculating a space frequency value of each of the plurality offrequency analysis regions; and merging frequency analysis regions eachwith a space frequency value conforming to a preset condition into thehigh-frequency region.

In some implementations, the preset condition includes: the spacefrequency value being greater than a preset frequency threshold.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to convert a part ofthe color-block image within the high-frequency region into a firstimage using a first interpolation algorithm by acts of: determiningwhether a color of a first simulation pixel is identical to that of anoriginal pixel at a same position as the first simulation pixel; whenthe color of the first simulation pixel is identical to that of theoriginal pixel at the same position as the first simulation pixel,determining a pixel value of the original pixel as a pixel value of thefirst simulation pixel; and when the color of the first simulation pixelis different from that of the original pixel at the same position as thefirst simulation pixel, determining the pixel value of the firstsimulation pixel according to a pixel value of an association pixel, inwhich the association pixel is selected from an image pixel unit with asame color as the first simulation pixel and adjacent to an image pixelunit comprising the original pixel.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to determine thepixel value of the first simulation pixel according to a pixel value ofan association pixel by acts of: calculating a change of the color ofthe first simulation pixel in each direction of at least two directionsaccording to the pixel value of the association pixel; calculating aweight in each direction of the at least two directions according to thechange; and calculating the pixel value of the first simulation pixelaccording to the weight and the pixel value of the association pixel.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to convert a part ofthe color-block image beyond the high-frequency region into a secondimage using a second interpolation algorithm by acts of: calculating anaverage pixel value of each image pixel unit of the color-block image;determining whether a color of a second simulation pixel is identical tothat of an original pixel at a same position as the second simulationpixel; when the color of the second simulation pixel is identical tothat of the original pixel at the same position as the second simulationpixel, determining an average pixel value of an image pixel unitcomprising the original pixel as a pixel value of the second simulationpixel; and when the color of the second simulation pixel is differentfrom that of the original pixel at the same position as the secondsimulation pixel, determining an average pixel value of an image pixelunit with a same color as the second simulation pixel and adjacent to animage pixel unit comprising the original pixel as a pixel value of thesecond simulation pixel.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to perform followingoperations: performing a white-balance compensation on the color-blockimage; and performing a reverse white-balance compensation on the firstimage.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to perform followingoperations: performing at least one of a bad-point compensation and acrosstalk compensation on the color-block image.

In some implementations, the processor 1002 is configured to run aprogram corresponding to the executable program codes by reading theexecutable program codes stored in the memory 1003, to perform followingoperations: performing at least one of a mirror shape correction, ademosaicking processing, a denoising processing and an edge sharpeningprocessing on the first image.

In some implementations, the electronic device may be electronicequipment provided with an image sensor, such as a mobile phone or atablet computer, which is not limited herein.

The electronic device 1000 may further include an inputting component(not illustrated in FIG. 25). It should be understood that, theinputting component may further include one or more of the followings:an inputting interface, a physical button of the electronic device 1000,a microphone, etc.

It should be understood that, the electronic device 1000 may furtherinclude one or more of the following components (not illustrated in FIG.25): an audio component, an input/output (I/O) interface, a sensorcomponent and a communication component. The audio component isconfigured to output and/or input audio signals, for example, the audiocomponent includes a microphone. The I/O interface is configured toprovide an interface between the processor 1002 and peripheral interfacemodules. The sensor component includes one or more sensors to providestatus assessments of various aspects of the electronic device 1000. Thecommunication component is configured to facilitate communication, wiredor wirelessly, between the electronic device 1000 and other devices.

Reference throughout this specification to “an embodiment,” “someembodiments,” “an example,” “a specific example,” or “some examples,”means that a particular feature, structure, material, or characteristicdescribed in connection with the embodiment or example is included in atleast one embodiment or example of the present disclosure. In thisspecification, exemplary descriptions of aforesaid terms are notnecessarily referring to the same embodiment or example. Furthermore,the particular features, structures, materials, or characteristics maybe combined in any suitable manner in one or more embodiments orexamples. Moreover, those skilled in the art could combine differentembodiments or different characteristics in embodiments or examplesdescribed in the present disclosure.

Any process or method described in a flow chart or described herein inother ways may be understood to include one or more modules, segments orportions of codes of executable instructions for achieving specificlogical functions or steps in the process, and the scope of a preferredembodiment of the present disclosure includes other implementations,wherein the order of execution may differ from that which is depicted ordiscussed, including according to involved function, executingconcurrently or with partial concurrence or in the contrary order toperform the function, which should be understood by those skilled in theart.

The logic and/or step described in other manners herein or shown in theflow chart, for example, a particular sequence table of executableinstructions for realizing the logical function, may be specificallyachieved in any computer readable medium to be used by the instructionexecution system, device or equipment (such as the system based oncomputers, the system comprising processors or other systems capable ofacquiring the instruction from the instruction execution system, deviceand equipment and executing the instruction), or to be used incombination with the instruction execution system, device and equipment.As to the specification, “the computer readable medium” may be anydevice adaptive for including, storing, communicating, propagating ortransferring programs to be used by or in combination with theinstruction execution system, device or equipment. More specificexamples of the computer-readable medium comprise but are not limitedto: an electronic connection (an electronic device) with one or morewires, a portable computer enclosure (a magnetic device), a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or a flash memory), an optical fiber device anda portable compact disk read-only memory (CDROM). In addition, thecomputer-readable medium may even be a paper or other appropriate mediumcapable of printing programs thereon, this is because, for example, thepaper or other appropriate medium may be optically scanned and thenedited, decrypted or processed with other appropriate methods whennecessary to obtain the programs in an electric manner, and then theprograms may be stored in the computer memories.

It should be understood that each part of the present disclosure may berealized by hardware, software, firmware or their combination. In theabove embodiments, a plurality of steps or methods may be realized bythe software or firmware stored in the memory and executed by theappropriate instruction execution system. For example, if it is realizedby the hardware, likewise in another embodiment, the steps or methodsmay be realized by one or a combination of the following techniquesknown in the art: a discrete logic circuit having a logic gate circuitfor realizing a logic function of a data signal, an application-specificintegrated circuit having an appropriate combination logic gate circuit,a programmable gate array (PGA), a field programmable gate array (FPGA),etc.

Those skilled in the art shall understand that all or parts of the stepsin the above exemplifying method for the present disclosure may beachieved by commanding the related hardware with programs, the programsmay be stored in a computer-readable storage medium, and the programscomprise one or a combination of the steps in the method embodiments ofthe present disclosure when running on a computer.

In addition, each function cell of the embodiments of the presentdisclosure may be integrated in a processing module, or these cells maybe separate physical existence, or two or more cells are integrated in aprocessing module. The integrated module may be realized in a form ofhardware or in a form of software function modules. When the integratedmodule is realized in a form of software function module and is sold orused as a standalone product, the integrated module may be stored in acomputer-readable storage medium.

The storage medium mentioned above may be read-only memories, magneticdisks, CD, etc.

Although embodiments of present disclosure have been shown and describedabove, it should be understood that above embodiments are justexplanatory, and cannot be construed to limit the present disclosure,for those skilled in the art, changes, alternatives, and modificationscan be made to the embodiments without departing from spirit, principlesand scope of the present disclosure.

What is claimed is:
 1. An image processing method, configured to processa color-block image output by an image sensor to output a simulationimage, wherein the image sensor comprises an array of photosensitivepixel units and an array of filter units arranged on the array ofphotosensitive pixel units, each filter unit corresponds to onephotosensitive pixel unit, each photosensitive pixel unit comprises aplurality of photosensitive pixels, the color-block image comprisesimage pixel units arranged in a preset array, each image pixel unitcomprises a plurality of original pixels, each photosensitive pixelcorresponds to one original pixel, and the image processing methodcomprises: dividing the color-block image into a plurality of frequencyanalysis regions; calculating a space frequency value of each of theplurality of frequency analysis regions; merging frequency analysisregions each with the space frequency value conforming to a presetcondition into the high-frequency region; converting the color-blockimage into the simulation image, wherein the simulation image comprisessimulation pixels arranged in a preset array, the simulation pixelcomprises a current pixel, the original pixel comprises an associationpixel corresponding to the current pixel, the converting the color-blockimage into the simulation image comprises: determining whether theassociation pixel is within the high-frequency region; when theassociation pixel is within the high-frequency region, determiningwhether a color of the current pixel is identical to that of theassociation pixel; when the color of the current pixel is identical tothat of the association pixel, determining a pixel value of theassociation pixel as a pixel value of the current pixel; when the colorof the current pixel is different from that of the association pixel,determining a pixel value of the current pixel according to a pixelvalue of an association pixel unit using a first interpolationalgorithm, wherein the image pixel unit comprises the association pixelunit, the association pixel unit comprises a plurality of originalpixels each with the same color as the current pixel and adjacent to theassociation pixel; when the association pixel is beyond thehigh-frequency region, calculating a pixel value of the current pixelusing a second interpolation algorithm, wherein, a complexity of thesecond interpolation algorithm is less than that of the firstinterpolation algorithm.
 2. The image processing method according toclaim 1, wherein the preset condition comprises: the space frequencyvalue being greater than a preset frequency threshold.
 3. The imageprocessing method according to claim 1, wherein the preset arraycomprises a Bayer array.
 4. The image processing method according toclaim 1, wherein the image pixel unit comprises original pixels arrangedin a 2-by-2 array.
 5. The image processing method according to claim 1,wherein the determining the pixel value of the current pixel accordingto the pixel value of the association pixel unit using the firstinterpolation algorithm, comprises: calculating a change of the color ofthe association pixel in each direction of at least two directions;calculating a weight of the association pixel in each direction of theat least two directions; and calculating the pixel value of the currentpixel according to the weight and the change.
 6. The image processingmethod according to claim 1, wherein, before the determining the pixelvalue of the current pixel according to the pixel value of theassociation pixel unit using the first interpolation algorithm, themethod further comprises: performing a white-balance compensation on thecolor-block image; wherein, after the determining the pixel value of thecurrent pixel according to the pixel value of the association pixel unitusing the first interpolation algorithm, the method further comprises:performing a reverse white-balance compensation on the simulation image.7. The image processing method according to claim 1, wherein, before thedetermining the pixel value of the current pixel according to the pixelvalue of the association pixel unit using the first interpolationalgorithm, the method further comprises: performing a bad-pointcompensation on the color-block image.
 8. The image processing methodaccording to claim 1, wherein, before the determining the pixel value ofthe current pixel according to the pixel value of the association pixelunit using the first interpolation algorithm, the method furthercomprises: performing a crosstalk compensation on the color-block image.9. The image processing method according to claim 1, wherein, after thedetermining the pixel value of the current pixel according to the pixelvalue of the association pixel unit using the first interpolationalgorithm, the method further comprises: performing a mirror shapecorrection, a demosaicking processing, a denoising processing and anedge sharpening processing on the simulation image.
 10. An imageprocessing apparatus, configured to process a color-block image outputby an image sensor to output a simulation image, wherein the imagesensor comprises an array of photosensitive pixel units and an array offilter units arranged on the array of photosensitive pixel units, eachfilter unit corresponds to one photosensitive pixel unit, and eachphotosensitive pixel unit comprises a plurality of photosensitivepixels, the color-block image comprises image pixel units arranged in apreset array, each image pixel unit comprises a plurality of originalpixels, each photosensitive pixel corresponds to one original pixel; theimage processing apparatus comprises: a non-transitory computer-readablemedium comprising computer-executable instructions stored thereon, andan instruction execution system which is configured by the instructionsto implement: a dividing module, configured to divide the color-blockimage into a plurality of frequency analysis regions; a calculatingmodule, configured to calculate a space frequency value of each of theplurality of frequency analysis regions; a merging module, configured tomerge frequency analysis regions each with the space frequency valueconforming to a preset condition into the high-frequency region; aconverting module, configured to convert the color-block image into thesimulation image, wherein the simulation image comprises simulationpixels arranged in a preset array, the simulation pixel comprises acurrent pixel, the original pixel comprises an association pixelcorresponding to the current simulation pixel; the converting modulecomprising: a first determining unit, configured to determine whetherthe association pixel is within the high-frequency region; a seconddetermining unit, configured to, when the association pixel is withinthe high-frequency region, determine whether a color of the currentpixel is identical to that of the association pixel; a first calculatingunit, configured to, when the color of the current pixel is identical tothat of the association pixel, determine a pixel value of theassociation pixel as a pixel value of the current pixel; a secondcalculating unit, configured to, when the color of the current pixel isdifferent from that of the association pixel, determine a pixel value ofthe current pixel according to a pixel value of an association pixelunit using a first interpolation algorithm, wherein the image pixel unitcomprises the association pixel unit, the association pixel unitcomprises a plurality of original pixels each with the same color as thecurrent pixel and adjacent to the association pixel; and a thirdcalculating unit, configured to, when the association pixel is beyondthe high-frequency region, calculate a pixel value of the current pixelusing a second interpolation algorithm, wherein, a complexity of thesecond interpolation algorithm is less than that of the firstinterpolation algorithm.
 11. The image processing apparatus according toclaim 10, wherein the preset condition comprises: the space frequencyvalue being greater than a preset frequency threshold.
 12. The imageprocessing apparatus according to claim 10, wherein the preset arraycomprises a Bayer array.
 13. The image processing apparatus according toclaim 10, wherein the image pixel unit comprises original pixelsarranged in a 2-by-2 array.
 14. The image processing apparatus accordingto claim 10, wherein the second calculating unit comprises: a firstcalculating sub unit, configured to calculate a change of the color ofthe association pixel in each direction of at least two directions; asecond calculating sub unit, configured to calculate a weight of theassociation pixel in each direction of the at least two directions; anda third calculating sub unit, configured to calculate the pixel value ofthe current pixel according to the weight and the change.
 15. The imageprocessing apparatus according to claim 10, wherein the convertingmodule comprises: a white-balance compensation unit, configured toperform a white-balance compensation on the color-block image; a reversewhite-balance compensation unit, configured to perform a reversewhite-balance compensation on the simulation image.
 16. The imageprocessing apparatus according to claim 10, wherein the convertingmodule comprises: a bad-point compensation unit, configured to perform abad-point compensation on the color-block image.
 17. The imageprocessing apparatus according to claim 10, wherein the convertingmodule comprises: a crosstalk compensation unit, configured to perform acrosstalk compensation on the color-block image.
 18. The imageprocessing apparatus according to claim 10, wherein the convertingmodule comprises: a processing unit, configured to perform a mirrorshape correction, a demosaicking processing, a denoising processing andan edge sharpening processing on the simulation image.
 19. An electronicdevice, comprising: an imaging device, wherein the imaging devicecomprises an image sensor and an image processing apparatus; and a touchscreen; wherein the image sensor is configured to output a color-blockimage, in which the image sensor comprises an array of photosensitivepixel units and an array of filter units arranged on the array ofphotosensitive pixel units, each filter unit corresponds to onephotosensitive pixel unit, each photosensitive pixel unit comprises aplurality of photosensitive pixels, the color-block image comprisesimage pixel units arranged in a preset array, each image pixel unitcomprises a plurality of original pixels, each photosensitive pixel unitcorresponds to one image pixel unit, each photosensitive pixelcorresponds to one original pixel; the image processing apparatuscomprises: a non-transitory computer-readable medium comprisingcomputer-executable instructions stored thereon, and an instructionexecution system which is configured by the instructions to implement: adividing module, configured to divide the color-block image into aplurality of frequency analysis regions; a calculating module,configured to calculate a space frequency value of each of the pluralityof frequency analysis regions; a merging module, configured to mergefrequency analysis regions each with the space frequency valueconforming to a preset condition into the high-frequency region; aconverting module, configured to convert the color-block image into thesimulation image, wherein the simulation image comprises simulationpixels arranged in a preset array, the simulation pixel comprises acurrent pixel, the original pixel comprises an association pixelcorresponding to the current simulation pixel; the converting modulecomprising: a first determining unit, configured to determine whetherthe association pixel is within the high-frequency region; a seconddetermining unit, configured to, when the association pixel is withinthe high-frequency region, determine whether a color of the currentpixel is identical to that of the association pixel; a first calculatingunit, configured to, when the color of the current pixel is identical tothat of the association pixel, determine a pixel value of theassociation pixel as a pixel value of the current pixel; a secondcalculating unit, configured to, when the color of the current pixel isdifferent from that of the association pixel, determine a pixel value ofthe current pixel according to a pixel value of an association pixelunit using a first interpolation algorithm, wherein the image pixel unitcomprises the association pixel unit, the association pixel unitcomprises a plurality of original pixels each with the same color as thecurrent pixel and adjacent to the association pixel; and a thirdcalculating unit, configured to, when the association pixel is beyondthe high-frequency region, calculate a pixel value of the current pixelusing a second interpolation algorithm, wherein, a complexity of thesecond interpolation algorithm is less than that of the firstinterpolation algorithm.
 20. The electronic device according to claim19, wherein the electronic device comprises a mobile phone or a tablet.